Computer networks typically include a service for logging event messages that are propagated through the network. Logging event messages may be useful for a variety of reasons including computer system management, network security analysis, and debugging. Syslog, detailed in Request For Comments (RFC) 3164, is one example of a data logging service which provides a standard for logging event messages in an IP network. Network devices, such as routers, workstations, and servers, within the computer network are examples of network nodes capable of logging and forwarding event messages (“syslog messages”). Generally, syslog messages are not stored by applications running on the devices within the network. Instead, these messages are sent to a dedicated server that saves the received messages into a log file or database. In most cases, acknowledgement or receipt of the message is not required or even desired.
When an event is generated by a device within the network, the device will send a small textual message to a syslog server. The syslog messages generally consist of identifiers for the facility that generated the message, message severity level, and timestamp indicating the time at which the event occurred. After receiving a message, the syslog server will store the message in a database.
As networks grow and become increasingly complex, the volume of structured events, such as syslog messages sent through the network, also increases. For large networks, it is not uncommon for a syslog database to store thousands of records per second continuously for weeks and months at a time. In many instances, these databases contain billions of records. Efficiently accessing, interpreting, and reporting against the data becomes a problem for databases of such enormity, as query performance tends to deteriorate as database table size increases.
For example, depending on the database vendor, insert and query operations show evident slowdown when a table holds 60 MB, or roughly 400,000 records on average. However, when a database contains millions or billions of records, data operations may take days to return.
The approaches described in this section are approaches that could be pursued, but not necessarily approaches that have been previously conceived or pursued. Therefore, unless otherwise indicated, it should not be assumed that any of the approaches described in this section qualify as prior art merely by virtue of their inclusion in this section.